{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* By: Illya Barziy\n",
    "* Email: illyabarziy@gmail.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RiskEstimators class functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This description is partially based on the __scikit-learn User Guide on Covariance estimation__ [available here](https://scikit-learn.org/stable/modules/covariance.html#robust-covariance)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Risk Estimators class includes the implementations of functions for different ways to calculate and adjust Covariance matrices.\n",
    "\n",
    "The following algorithms are now implemented:\n",
    "- Minimum Covariance Determinant\n",
    "- Maximum likelihood covariance estimator (Empirical covariance)\n",
    "- Covariance estimator with shrinkage\n",
    "  - Basic shrinkage\n",
    "  - Ledoit-Wolf shrinkage\n",
    "  - Oracle Approximating Shrinkage\n",
    "- Semi-Covariance matrix\n",
    "- Exponentially-weighted Covariance matrix\n",
    "- De-noising and De-toning Covariance/Correlation Matrix\n",
    "  - Constant Residual Eigenvalue De-noising Method\n",
    "  - Targeted Shrinkage De-noising Method\n",
    "  - De-toning\n",
    "- Transforming covariance matrix to correlation matrix and back\n",
    "\n",
    "This Notebook will describe the above algorithms as well as provide use cases and analysis of results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Minimum Covariance Determinant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the __scikit-learn User Guide on Covariance estimation__:\n",
    "\n",
    "_\"The outliers are appearing in real data sets and seriously affect the Empirical covariance estimator and the Covariance estimators with shrinkage. For this reason, a robust covariance estimator is needed in order to discard/downweight the outliers in the data\"._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The robust estimator presented in the package is the Minimum Covariance Determinant estimator, introduced by P.J. Rousseeuw.\n",
    "\n",
    "_\"The basic idea of the algorithm is to find a set of observations that are not outliers and compute their empirical covariance matrix, which is then rescaled to compensate for the performed selection of observations\"._\n",
    "\n",
    "Our function is a wrap around the sklearn's MinCovDet class, which uses FastMCD algorithm, developed by Rousseeuw and Van Driessen.\n",
    "\n",
    "A detailed description of the algorithm is available in the paper by _Mia Hubert_ and _Michiel Debruyne_ __Minimum covariance determinant__ [available here](https://wis.kuleuven.be/stat/robust/papers/2010/wire-mcd.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples of use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can calculate the Minimum Covariance Determinant estimator of covariance for a data set of stock prices and compare it to the simple covariance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mlfinlab as ml\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "      <th>IEF</th>\n",
       "      <th>EWQ</th>\n",
       "      <th>EWU</th>\n",
       "      <th>XLB</th>\n",
       "      <th>XLE</th>\n",
       "      <th>...</th>\n",
       "      <th>XLU</th>\n",
       "      <th>EPP</th>\n",
       "      <th>FXI</th>\n",
       "      <th>VGK</th>\n",
       "      <th>VPL</th>\n",
       "      <th>SPY</th>\n",
       "      <th>TLT</th>\n",
       "      <th>BND</th>\n",
       "      <th>CSJ</th>\n",
       "      <th>DIA</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2008-01-02</th>\n",
       "      <td>49.273335</td>\n",
       "      <td>35.389999</td>\n",
       "      <td>106.639999</td>\n",
       "      <td>52.919998</td>\n",
       "      <td>78.220001</td>\n",
       "      <td>87.629997</td>\n",
       "      <td>37.939999</td>\n",
       "      <td>47.759998</td>\n",
       "      <td>41.299999</td>\n",
       "      <td>79.500000</td>\n",
       "      <td>...</td>\n",
       "      <td>42.090000</td>\n",
       "      <td>51.173328</td>\n",
       "      <td>55.983330</td>\n",
       "      <td>74.529999</td>\n",
       "      <td>67.309998</td>\n",
       "      <td>144.929993</td>\n",
       "      <td>94.379997</td>\n",
       "      <td>77.360001</td>\n",
       "      <td>101.400002</td>\n",
       "      <td>130.630005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-03</th>\n",
       "      <td>49.716667</td>\n",
       "      <td>35.290001</td>\n",
       "      <td>107.000000</td>\n",
       "      <td>53.119999</td>\n",
       "      <td>78.349998</td>\n",
       "      <td>87.809998</td>\n",
       "      <td>37.919998</td>\n",
       "      <td>48.060001</td>\n",
       "      <td>42.049999</td>\n",
       "      <td>80.440002</td>\n",
       "      <td>...</td>\n",
       "      <td>42.029999</td>\n",
       "      <td>51.293331</td>\n",
       "      <td>55.599998</td>\n",
       "      <td>74.800003</td>\n",
       "      <td>67.500000</td>\n",
       "      <td>144.860001</td>\n",
       "      <td>94.250000</td>\n",
       "      <td>77.459999</td>\n",
       "      <td>101.519997</td>\n",
       "      <td>130.740005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-04</th>\n",
       "      <td>48.223331</td>\n",
       "      <td>34.599998</td>\n",
       "      <td>106.970001</td>\n",
       "      <td>51.759998</td>\n",
       "      <td>76.570000</td>\n",
       "      <td>88.040001</td>\n",
       "      <td>36.990002</td>\n",
       "      <td>46.919998</td>\n",
       "      <td>40.779999</td>\n",
       "      <td>77.500000</td>\n",
       "      <td>...</td>\n",
       "      <td>42.349998</td>\n",
       "      <td>49.849998</td>\n",
       "      <td>54.536671</td>\n",
       "      <td>72.980003</td>\n",
       "      <td>65.769997</td>\n",
       "      <td>141.309998</td>\n",
       "      <td>94.269997</td>\n",
       "      <td>77.550003</td>\n",
       "      <td>101.650002</td>\n",
       "      <td>128.169998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-07</th>\n",
       "      <td>48.576668</td>\n",
       "      <td>34.630001</td>\n",
       "      <td>106.949997</td>\n",
       "      <td>51.439999</td>\n",
       "      <td>76.650002</td>\n",
       "      <td>88.199997</td>\n",
       "      <td>37.259998</td>\n",
       "      <td>47.060001</td>\n",
       "      <td>40.220001</td>\n",
       "      <td>77.199997</td>\n",
       "      <td>...</td>\n",
       "      <td>43.230000</td>\n",
       "      <td>50.416672</td>\n",
       "      <td>56.116669</td>\n",
       "      <td>72.949997</td>\n",
       "      <td>65.650002</td>\n",
       "      <td>141.190002</td>\n",
       "      <td>94.680000</td>\n",
       "      <td>77.570000</td>\n",
       "      <td>101.720001</td>\n",
       "      <td>128.059998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-08</th>\n",
       "      <td>48.200001</td>\n",
       "      <td>34.389999</td>\n",
       "      <td>107.029999</td>\n",
       "      <td>51.320000</td>\n",
       "      <td>76.220001</td>\n",
       "      <td>88.389999</td>\n",
       "      <td>36.970001</td>\n",
       "      <td>46.400002</td>\n",
       "      <td>39.599998</td>\n",
       "      <td>75.849998</td>\n",
       "      <td>...</td>\n",
       "      <td>43.240002</td>\n",
       "      <td>49.566669</td>\n",
       "      <td>55.326672</td>\n",
       "      <td>72.400002</td>\n",
       "      <td>65.360001</td>\n",
       "      <td>138.910004</td>\n",
       "      <td>94.570000</td>\n",
       "      <td>77.650002</td>\n",
       "      <td>101.739998</td>\n",
       "      <td>125.849998</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                  EEM        EWG         TIP        EWJ        EFA        IEF  \\\n",
       "Date                                                                            \n",
       "2008-01-02  49.273335  35.389999  106.639999  52.919998  78.220001  87.629997   \n",
       "2008-01-03  49.716667  35.290001  107.000000  53.119999  78.349998  87.809998   \n",
       "2008-01-04  48.223331  34.599998  106.970001  51.759998  76.570000  88.040001   \n",
       "2008-01-07  48.576668  34.630001  106.949997  51.439999  76.650002  88.199997   \n",
       "2008-01-08  48.200001  34.389999  107.029999  51.320000  76.220001  88.389999   \n",
       "\n",
       "                  EWQ        EWU        XLB        XLE  ...        XLU  \\\n",
       "Date                                                    ...              \n",
       "2008-01-02  37.939999  47.759998  41.299999  79.500000  ...  42.090000   \n",
       "2008-01-03  37.919998  48.060001  42.049999  80.440002  ...  42.029999   \n",
       "2008-01-04  36.990002  46.919998  40.779999  77.500000  ...  42.349998   \n",
       "2008-01-07  37.259998  47.060001  40.220001  77.199997  ...  43.230000   \n",
       "2008-01-08  36.970001  46.400002  39.599998  75.849998  ...  43.240002   \n",
       "\n",
       "                  EPP        FXI        VGK        VPL         SPY        TLT  \\\n",
       "Date                                                                            \n",
       "2008-01-02  51.173328  55.983330  74.529999  67.309998  144.929993  94.379997   \n",
       "2008-01-03  51.293331  55.599998  74.800003  67.500000  144.860001  94.250000   \n",
       "2008-01-04  49.849998  54.536671  72.980003  65.769997  141.309998  94.269997   \n",
       "2008-01-07  50.416672  56.116669  72.949997  65.650002  141.190002  94.680000   \n",
       "2008-01-08  49.566669  55.326672  72.400002  65.360001  138.910004  94.570000   \n",
       "\n",
       "                  BND         CSJ         DIA  \n",
       "Date                                           \n",
       "2008-01-02  77.360001  101.400002  130.630005  \n",
       "2008-01-03  77.459999  101.519997  130.740005  \n",
       "2008-01-04  77.550003  101.650002  128.169998  \n",
       "2008-01-07  77.570000  101.720001  128.059998  \n",
       "2008-01-08  77.650002  101.739998  125.849998  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Getting the data\n",
    "stock_prices = pd.read_csv('../../Sample-Data/stock_prices.csv', parse_dates=True, index_col='Date')\n",
    "stock_prices = stock_prices.dropna(axis=1)\n",
    "stock_prices.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2008-01-02</th>\n",
       "      <td>49.273335</td>\n",
       "      <td>35.389999</td>\n",
       "      <td>106.639999</td>\n",
       "      <td>52.919998</td>\n",
       "      <td>78.220001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-03</th>\n",
       "      <td>49.716667</td>\n",
       "      <td>35.290001</td>\n",
       "      <td>107.000000</td>\n",
       "      <td>53.119999</td>\n",
       "      <td>78.349998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-04</th>\n",
       "      <td>48.223331</td>\n",
       "      <td>34.599998</td>\n",
       "      <td>106.970001</td>\n",
       "      <td>51.759998</td>\n",
       "      <td>76.570000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-07</th>\n",
       "      <td>48.576668</td>\n",
       "      <td>34.630001</td>\n",
       "      <td>106.949997</td>\n",
       "      <td>51.439999</td>\n",
       "      <td>76.650002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-01-08</th>\n",
       "      <td>48.200001</td>\n",
       "      <td>34.389999</td>\n",
       "      <td>107.029999</td>\n",
       "      <td>51.320000</td>\n",
       "      <td>76.220001</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  EEM        EWG         TIP        EWJ        EFA\n",
       "Date                                                              \n",
       "2008-01-02  49.273335  35.389999  106.639999  52.919998  78.220001\n",
       "2008-01-03  49.716667  35.290001  107.000000  53.119999  78.349998\n",
       "2008-01-04  48.223331  34.599998  106.970001  51.759998  76.570000\n",
       "2008-01-07  48.576668  34.630001  106.949997  51.439999  76.650002\n",
       "2008-01-08  48.200001  34.389999  107.029999  51.320000  76.220001"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Leaving only 5 stocks in the dataset, so the differences between the \n",
    "# calculated covariance matrices would be easy to observe.\n",
    "stock_prices = stock_prices.iloc[:, :5]\n",
    "stock_prices.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Minimum Covariance Determinant estimator is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000146</td>\n",
       "      <td>0.000112</td>\n",
       "      <td>-0.000005</td>\n",
       "      <td>0.000076</td>\n",
       "      <td>0.000102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000112</td>\n",
       "      <td>0.000154</td>\n",
       "      <td>-0.000007</td>\n",
       "      <td>0.000076</td>\n",
       "      <td>0.000114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000005</td>\n",
       "      <td>-0.000007</td>\n",
       "      <td>0.000011</td>\n",
       "      <td>-0.000003</td>\n",
       "      <td>-0.000005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000076</td>\n",
       "      <td>0.000076</td>\n",
       "      <td>-0.000003</td>\n",
       "      <td>0.000098</td>\n",
       "      <td>0.000077</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000102</td>\n",
       "      <td>0.000114</td>\n",
       "      <td>-0.000005</td>\n",
       "      <td>0.000077</td>\n",
       "      <td>0.000100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000146  0.000112 -0.000005  0.000076  0.000102\n",
       "EWG  0.000112  0.000154 -0.000007  0.000076  0.000114\n",
       "TIP -0.000005 -0.000007  0.000011 -0.000003 -0.000005\n",
       "EWJ  0.000076  0.000076 -0.000003  0.000098  0.000077\n",
       "EFA  0.000102  0.000114 -0.000005  0.000077  0.000100"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A class that has the Minimum Covariance Determinant estimator\n",
    "risk_estimators = ml.portfolio_optimization.RiskEstimators()\n",
    "\n",
    "# Finding the Minimum Covariance Determinant estimator on price data and with set random seed to 0\n",
    "min_cov_det = risk_estimators.minimum_covariance_determinant(stock_prices, price_data=True, random_state=0)\n",
    "\n",
    "# For the simple covariance, we need to transform the stock prices to returns\n",
    "\n",
    "# A class with function to calculate returns from prices\n",
    "returns_estimation = ml.portfolio_optimization.ReturnsEstimators()\n",
    "\n",
    "# Calcualting the data set of returns\n",
    "stock_returns = returns_estimation.calculate_returns(stock_prices)\n",
    "\n",
    "# Finding the simple covariance matrix from a series of returns\n",
    "cov_matrix = stock_returns.cov()\n",
    "\n",
    "# Transforming Minimum Covariance Determinant estimator from np.array to pd.DataFrame\n",
    "min_cov_det = pd.DataFrame(min_cov_det, index=cov_matrix.index, columns=cov_matrix.columns)\n",
    "\n",
    "print('The Minimum Covariance Determinant estimator is:')\n",
    "min_cov_det"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Simple Covariance is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000466</td>\n",
       "      <td>0.000350</td>\n",
       "      <td>-0.000017</td>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000350</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>0.000303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000017</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>-0.000012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>0.000232</td>\n",
       "      <td>0.000218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000324</td>\n",
       "      <td>0.000303</td>\n",
       "      <td>-0.000012</td>\n",
       "      <td>0.000218</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000466  0.000350 -0.000017  0.000255  0.000324\n",
       "EWG  0.000350  0.000372 -0.000015  0.000221  0.000303\n",
       "TIP -0.000017 -0.000015  0.000019 -0.000009 -0.000012\n",
       "EWJ  0.000255  0.000221 -0.000009  0.000232  0.000218\n",
       "EFA  0.000324  0.000303 -0.000012  0.000218  0.000278"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('The Simple Covariance is:')\n",
    "cov_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the results, the absolute values in the Minimum Covariance Determinant estimator are lower in comparison to the simple Covariance matrix, which means that the algorithm has eliminated some of the outliers in the data and the resulting covariance matrix estimator is a more robust one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Maximum likelihood covariance estimator (Empirical covariance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the __scikit-learn User Guide on Covariance estimation__:\n",
    "\n",
    "_\"The covariance matrix of a data set can be well approximated by the maximum likelihood estimator (Empirical covariance) if the number of observations is big enough in relation to the number of features\"._\n",
    "\n",
    "_\"The Maximum Likelihood Estimator of a sample is an unbiased estimator of the corresponding population’s covariance matrix\"._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples of use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can calculate the Empirical covariance for a data set of stock prices and compare it to the simple covariance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Empirical Covariance is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000466</td>\n",
       "      <td>0.000350</td>\n",
       "      <td>-0.000017</td>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000350</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>0.000303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000017</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>-0.000012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>0.000232</td>\n",
       "      <td>0.000218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000324</td>\n",
       "      <td>0.000303</td>\n",
       "      <td>-0.000012</td>\n",
       "      <td>0.000218</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000466  0.000350 -0.000017  0.000255  0.000324\n",
       "EWG  0.000350  0.000372 -0.000015  0.000221  0.000303\n",
       "TIP -0.000017 -0.000015  0.000019 -0.000009 -0.000012\n",
       "EWJ  0.000255  0.000221 -0.000009  0.000232  0.000218\n",
       "EFA  0.000324  0.000303 -0.000012  0.000218  0.000278"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Finding the Empirical Covariance on price data\n",
    "empirical_cov = risk_estimators.empirical_covariance(stock_prices, price_data=True)\n",
    "\n",
    "# Transforming Empirical Covariance from np.array to pd.DataFrame\n",
    "empirical_cov = pd.DataFrame(empirical_cov, index=cov_matrix.index, columns=cov_matrix.columns)\n",
    "\n",
    "print('The Empirical Covariance is:')\n",
    "empirical_cov"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Simple Covariance is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000466</td>\n",
       "      <td>0.000350</td>\n",
       "      <td>-0.000017</td>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000350</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>0.000303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000017</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>-0.000012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>0.000232</td>\n",
       "      <td>0.000218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000324</td>\n",
       "      <td>0.000303</td>\n",
       "      <td>-0.000012</td>\n",
       "      <td>0.000218</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000466  0.000350 -0.000017  0.000255  0.000324\n",
       "EWG  0.000350  0.000372 -0.000015  0.000221  0.000303\n",
       "TIP -0.000017 -0.000015  0.000019 -0.000009 -0.000012\n",
       "EWJ  0.000255  0.000221 -0.000009  0.000232  0.000218\n",
       "EFA  0.000324  0.000303 -0.000012  0.000218  0.000278"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('The Simple Covariance is:')\n",
    "cov_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is the same as from the standard covariance function from the pandas package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Covariance estimator with shrinkage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the __scikit-learn User Guide on Covariance estimation__:\n",
    "\n",
    "_\"The Maximum Likelihood Estimator is not a good estimator of the eigenvalues of the covariance matrix and the inverted matrix is not accurate. Sometimes, ... it cannot be inverted for numerical reasons\"._\n",
    "\n",
    "_\"To avoid problems with inversion, a transformation of the empirical covariance matrix has been introduced: the shrinkage\"._\n",
    "\n",
    "_\"Mathematically, this shrinkage consists in reducing the ratio between the smallest and the largest eigenvalues of the empirical covariance matrix\"._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic shrinkage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_\"This shrinkage is done by shifting every eigenvalue according to a given offset, which is equivalent to finding the l2-penalized Maximum Likelihood Estimator of the covariance matrix\"._\n",
    "\n",
    "_\"Shrinkage boils down to a simple a convex transformation\":_\n",
    "\n",
    "$$\\sum_{shrunk} = (1 - \\alpha)\\sum_{unshrunk} + \\alpha\\frac{Tr \\sum_{unshrunk}}{p}Id$$\n",
    "\n",
    "_\"The amount of shrinkage $\\alpha$ is setting a trade-off between bias and variance\"._\n",
    "\n",
    "In the implementation, $\\alpha$ is passed to a function as the $basic\\_shrinkage$ parameter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ledoit-Wolf shrinkage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_\"The Ledoit-Wolf shrinkage is based on computing the optimal shrinkage coefficient $\\alpha$ that minimizes the Mean Squared Error between the estimated and the real covariance matrix\"._\n",
    "\n",
    "The algorithm is described in more detail in the paper by _Olivier Ledoit_ and _Michael Wolf_ __A well-conditioned estimator forlarge-dimensional covariance matrices__ [available here](http://perso.ens-lyon.fr/patrick.flandrin/LedoitWolf_JMA2004.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Oracle Approximating shrinkage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_\"Assuming that the data are Gaussian distributed, Chen et al. derived a formula aimed at choosing a shrinkage coefficient $\\alpha$ that yields a smaller Mean Squared Error than the one given by Ledoit and Wolf’s formula\"._\n",
    "\n",
    "_\"The resulting estimator is known as the Oracle Shrinkage Approximating estimator of the covariance\"._\n",
    "\n",
    "The algorithm is described in more detail in the paper by _Y. Chen_, _A. Wiesel_, _Y.C. Eldar_ and _A.O. Hero_ __Shrinkage Algorithms for MMSE Covariance Estimation__ [available here](https://webee.technion.ac.il/people/YoninaEldar/104.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples of use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can calculate the Shrinked Covariances for every method and compare them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Basic Shrinked covariance with an alpha of 0.1 is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000446</td>\n",
       "      <td>0.000315</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000230</td>\n",
       "      <td>0.000292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000315</td>\n",
       "      <td>0.000362</td>\n",
       "      <td>-0.000013</td>\n",
       "      <td>0.000199</td>\n",
       "      <td>0.000273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000015</td>\n",
       "      <td>-0.000013</td>\n",
       "      <td>0.000045</td>\n",
       "      <td>-0.000008</td>\n",
       "      <td>-0.000011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000230</td>\n",
       "      <td>0.000199</td>\n",
       "      <td>-0.000008</td>\n",
       "      <td>0.000236</td>\n",
       "      <td>0.000197</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000292</td>\n",
       "      <td>0.000273</td>\n",
       "      <td>-0.000011</td>\n",
       "      <td>0.000197</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000446  0.000315 -0.000015  0.000230  0.000292\n",
       "EWG  0.000315  0.000362 -0.000013  0.000199  0.000273\n",
       "TIP -0.000015 -0.000013  0.000045 -0.000008 -0.000011\n",
       "EWJ  0.000230  0.000199 -0.000008  0.000236  0.000197\n",
       "EFA  0.000292  0.000273 -0.000011  0.000197  0.000278"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Finding the Shrinked Covariances on price data with every method\n",
    "shrinked_cov = risk_estimators.shrinked_covariance(stock_prices, price_data=True,\n",
    "                                                   shrinkage_type='all', basic_shrinkage=0.1)\n",
    "\n",
    "# Separating the Shrinked covariances for every method\n",
    "shrinked_cov_basic, shrinked_cov_lw, shrinked_cov_oas = shrinked_cov\n",
    "\n",
    "# Transforming each Shrinked Covariance from np.array to pd.DataFrame\n",
    "shrinked_cov_basic = pd.DataFrame(shrinked_cov_basic, index=cov_matrix.index, columns=cov_matrix.columns)\n",
    "shrinked_cov_lw = pd.DataFrame(shrinked_cov_lw, index=cov_matrix.index, columns=cov_matrix.columns)\n",
    "shrinked_cov_oas = pd.DataFrame(shrinked_cov_oas, index=cov_matrix.index, columns=cov_matrix.columns)\n",
    "\n",
    "print('The Basic Shrinked covariance with an alpha of 0.1 is:')\n",
    "shrinked_cov_basic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Ledoit-Wolf Shrinked covariance is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000464</td>\n",
       "      <td>0.000346</td>\n",
       "      <td>-0.000016</td>\n",
       "      <td>0.000252</td>\n",
       "      <td>0.000321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000346</td>\n",
       "      <td>0.000371</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000219</td>\n",
       "      <td>0.000300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000016</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000022</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>-0.000012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000252</td>\n",
       "      <td>0.000219</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>0.000233</td>\n",
       "      <td>0.000216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000321</td>\n",
       "      <td>0.000300</td>\n",
       "      <td>-0.000012</td>\n",
       "      <td>0.000216</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000464  0.000346 -0.000016  0.000252  0.000321\n",
       "EWG  0.000346  0.000371 -0.000015  0.000219  0.000300\n",
       "TIP -0.000016 -0.000015  0.000022 -0.000009 -0.000012\n",
       "EWJ  0.000252  0.000219 -0.000009  0.000233  0.000216\n",
       "EFA  0.000321  0.000300 -0.000012  0.000216  0.000278"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('The Ledoit-Wolf Shrinked covariance is:')\n",
    "shrinked_cov_lw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Oracle Approximating Shrinked covariance is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000465</td>\n",
       "      <td>0.000349</td>\n",
       "      <td>-0.000017</td>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000349</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>0.000303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000017</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>-0.000012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>0.000232</td>\n",
       "      <td>0.000218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000324</td>\n",
       "      <td>0.000303</td>\n",
       "      <td>-0.000012</td>\n",
       "      <td>0.000218</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000465  0.000349 -0.000017  0.000255  0.000324\n",
       "EWG  0.000349  0.000372 -0.000015  0.000221  0.000303\n",
       "TIP -0.000017 -0.000015  0.000020 -0.000009 -0.000012\n",
       "EWJ  0.000255  0.000221 -0.000009  0.000232  0.000218\n",
       "EFA  0.000324  0.000303 -0.000012  0.000218  0.000278"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('The Oracle Approximating Shrinked covariance is:')\n",
    "shrinked_cov_oas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Simple Covariance is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000466</td>\n",
       "      <td>0.000350</td>\n",
       "      <td>-0.000017</td>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000350</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>0.000303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000017</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>-0.000012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>0.000232</td>\n",
       "      <td>0.000218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000324</td>\n",
       "      <td>0.000303</td>\n",
       "      <td>-0.000012</td>\n",
       "      <td>0.000218</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000466  0.000350 -0.000017  0.000255  0.000324\n",
       "EWG  0.000350  0.000372 -0.000015  0.000221  0.000303\n",
       "TIP -0.000017 -0.000015  0.000019 -0.000009 -0.000012\n",
       "EWJ  0.000255  0.000221 -0.000009  0.000232  0.000218\n",
       "EFA  0.000324  0.000303 -0.000012  0.000218  0.000278"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('The Simple Covariance is:')\n",
    "cov_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Shrinked Covariance matrices for the Ledoit-Wolf and Oracle Approximating algorithms are similar with absolute covariance values in the Oracle Approximating covariance matrix being slightly bigger. With the basic Shrinkage covariance matrix with $\\alpha = 0.1$, the absolute values are even smaller. The Simple Covariance matrix has the highest absolute values in comparison."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Semi-Covariance matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Semi-covariance matrix is the way to measure the volatility of the negative returns or returns below a certain threshold. \n",
    "\n",
    "This measure can be used to decrease the negative volatility and is being more precise for this goal than the covariance matrix that measures both positive and negative variance. \n",
    "\n",
    "According to the __Minimum Downside Volatility Indices__ paper:\n",
    "\n",
    "_\"Each element in the Semi-Covariance matrix is calculated as:\"_\n",
    "\n",
    "$$SemiCov_{ij} = \\frac{1}{T}\\sum_{t=1}^{T}[Min(R_{i,t}-B,0)*Min(R_{j,t}-B,0)]$$\n",
    "\n",
    "_where $T$ is the number of observations,_ $R_{i,t}$ _is the return of an asset $i$ at time $t$, and $B$ is the threshold return._\n",
    "\n",
    "_If the $B$ is set to zero, the volatility of negative returns is measured._\n",
    "\n",
    "A deeper analysis of use cases of Semi-Covariance matrix is available in the paper by _Solactive AG - German Index Engineering_ __Minimum Downside Volatility Indices__ [available here](https://www.solactive.com/wp-content/uploads/2018/04/Solactive_Minimum-Downside-Volatility-Indices.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples of use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can calculate the Semi-Covariance and compare it to the simple covariance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Semi-Covariance matrix is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000044</td>\n",
       "      <td>0.000035</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.000025</td>\n",
       "      <td>0.000032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000035</td>\n",
       "      <td>0.000038</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.000023</td>\n",
       "      <td>0.000031</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.000002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000025</td>\n",
       "      <td>0.000023</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.000023</td>\n",
       "      <td>0.000022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000032</td>\n",
       "      <td>0.000031</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.000022</td>\n",
       "      <td>0.000029</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000044  0.000035  0.000002  0.000025  0.000032\n",
       "EWG  0.000035  0.000038  0.000002  0.000023  0.000031\n",
       "TIP  0.000002  0.000002  0.000002  0.000002  0.000002\n",
       "EWJ  0.000025  0.000023  0.000002  0.000023  0.000022\n",
       "EFA  0.000032  0.000031  0.000002  0.000022  0.000029"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Finding the Semi-Covariance on price data\n",
    "semi_cov = risk_estimators.semi_covariance(stock_prices, price_data=True, threshold_return=0)\n",
    "\n",
    "# Transforming Semi-Covariance from np.array to pd.DataFrame\n",
    "semi_cov = pd.DataFrame(semi_cov, index=cov_matrix.index, columns=cov_matrix.columns)\n",
    "\n",
    "print('The Semi-Covariance matrix is:')\n",
    "semi_cov"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Simple Covariance is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000466</td>\n",
       "      <td>0.000350</td>\n",
       "      <td>-0.000017</td>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000350</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>0.000303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000017</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>-0.000012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>0.000232</td>\n",
       "      <td>0.000218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000324</td>\n",
       "      <td>0.000303</td>\n",
       "      <td>-0.000012</td>\n",
       "      <td>0.000218</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000466  0.000350 -0.000017  0.000255  0.000324\n",
       "EWG  0.000350  0.000372 -0.000015  0.000221  0.000303\n",
       "TIP -0.000017 -0.000015  0.000019 -0.000009 -0.000012\n",
       "EWJ  0.000255  0.000221 -0.000009  0.000232  0.000218\n",
       "EFA  0.000324  0.000303 -0.000012  0.000218  0.000278"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('The Simple Covariance is:')\n",
    "cov_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the computation of the Semi-Covariance matrix is different from the usual computation of the covariance matrix, the absolute values in the Semi-Covariance matrix are significantly lower. Since it's a measure, let's multiply the Semi-Covariance matrix by 10 to better see the changes in the measures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Semi-Covariance matrix multiplied by 10 is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000438</td>\n",
       "      <td>0.000351</td>\n",
       "      <td>0.000018</td>\n",
       "      <td>0.000251</td>\n",
       "      <td>0.000322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000351</td>\n",
       "      <td>0.000377</td>\n",
       "      <td>0.000017</td>\n",
       "      <td>0.000231</td>\n",
       "      <td>0.000312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>0.000018</td>\n",
       "      <td>0.000017</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000016</td>\n",
       "      <td>0.000015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000251</td>\n",
       "      <td>0.000231</td>\n",
       "      <td>0.000016</td>\n",
       "      <td>0.000230</td>\n",
       "      <td>0.000222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000322</td>\n",
       "      <td>0.000312</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>0.000222</td>\n",
       "      <td>0.000285</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000438  0.000351  0.000018  0.000251  0.000322\n",
       "EWG  0.000351  0.000377  0.000017  0.000231  0.000312\n",
       "TIP  0.000018  0.000017  0.000019  0.000016  0.000015\n",
       "EWJ  0.000251  0.000231  0.000016  0.000230  0.000222\n",
       "EFA  0.000322  0.000312  0.000015  0.000222  0.000285"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('The Semi-Covariance matrix multiplied by 10 is:')\n",
    "semi_cov * 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see that the values in the two matrices are similar, however, some differences are present.\n",
    "\n",
    "For example, the simple Covariance between the EEM and TIP is negative, but the negative returns have positive covariance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exponentially-weighted Covariance matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each element in the Exponentially-weighted Covariance matrix is calculated as follows.\n",
    "\n",
    "First, we calculate the series of covariances for every observation time $t$ between each two elements $i$ and $j$:\n",
    "\n",
    "$$CovarSeries_{i,j}^{t} = (R_{i}^{t} - Mean(R_{i})) * (R_{j}^{t} - Mean(R_{j}))$$\n",
    "\n",
    "Then we apply the exponential weighted moving average based on the obtained series with decay in terms of span, as $\\alpha=\\frac{2}{span+1}$, for $span \\ge 1$\n",
    "\n",
    "$$ExponentialCovariance_{i,j} = ExponentialWeightedMovingAverage(CovarSeries_{i,j})[T]$$\n",
    "\n",
    "So, it's the last element from an exponentially weighted moving average series based on a series of covariances between returns of the corresponding assets. \n",
    "\n",
    "It is used to give greater weight to most relevant observations in computing the covariance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples of use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can calculate the Exponential Covariance and compare it to the simple covariance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Exponential Covariance matrix is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000282</td>\n",
       "      <td>0.000322</td>\n",
       "      <td>-0.000004</td>\n",
       "      <td>0.000190</td>\n",
       "      <td>0.000303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000322</td>\n",
       "      <td>0.000459</td>\n",
       "      <td>-0.000019</td>\n",
       "      <td>0.000237</td>\n",
       "      <td>0.000410</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000004</td>\n",
       "      <td>-0.000019</td>\n",
       "      <td>0.000009</td>\n",
       "      <td>-0.000011</td>\n",
       "      <td>-0.000016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000190</td>\n",
       "      <td>0.000237</td>\n",
       "      <td>-0.000011</td>\n",
       "      <td>0.000199</td>\n",
       "      <td>0.000229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000303</td>\n",
       "      <td>0.000410</td>\n",
       "      <td>-0.000016</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>0.000380</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000282  0.000322 -0.000004  0.000190  0.000303\n",
       "EWG  0.000322  0.000459 -0.000019  0.000237  0.000410\n",
       "TIP -0.000004 -0.000019  0.000009 -0.000011 -0.000016\n",
       "EWJ  0.000190  0.000237 -0.000011  0.000199  0.000229\n",
       "EFA  0.000303  0.000410 -0.000016  0.000229  0.000380"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Finding the Exponential Covariance on price data and span of 60\n",
    "exponential_cov = risk_estimators.exponential_covariance(stock_prices, price_data=True, window_span=60)\n",
    "\n",
    "# Transforming Semi-Covariance from np.array to pd.DataFrame\n",
    "exponential_cov = pd.DataFrame(exponential_cov, index=cov_matrix.index, columns=cov_matrix.columns)\n",
    "\n",
    "print('The Exponential Covariance matrix is:')\n",
    "exponential_cov"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Simple Covariance is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000466</td>\n",
       "      <td>0.000350</td>\n",
       "      <td>-0.000017</td>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000350</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>0.000303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000017</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>-0.000012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>0.000232</td>\n",
       "      <td>0.000218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000324</td>\n",
       "      <td>0.000303</td>\n",
       "      <td>-0.000012</td>\n",
       "      <td>0.000218</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000466  0.000350 -0.000017  0.000255  0.000324\n",
       "EWG  0.000350  0.000372 -0.000015  0.000221  0.000303\n",
       "TIP -0.000017 -0.000015  0.000019 -0.000009 -0.000012\n",
       "EWJ  0.000255  0.000221 -0.000009  0.000232  0.000218\n",
       "EFA  0.000324  0.000303 -0.000012  0.000218  0.000278"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('The Simple Covariance is:')\n",
    "cov_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the results it's seen that the variance of the EWG has increased in the last observations, whereas the the variance of the EEM has decreased. The covariance between the EEM and EWJ has decreased in the last observations.\n",
    "\n",
    "So, the covariance with higher weights in the most recent observations can be analyzed in comparison to covariance with equal weights (simple covariance). And the conclusions about how the covariance has changed over time can be drawn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## De-noising and De-toning Covariance/Correlation Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Constant Residual Eigenvalue De-noising Method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main idea behind de-noising the covariance matrix is to eliminate the eigenvalues of the covariance matrix that are representing noise and not useful information. \n",
    "\n",
    "This is done by determining the maximum theoretical value of the eigenvalue of such matrix as a threshold and then setting all the calculated eigenvalues below the threshold to the same value.\n",
    "\n",
    "The function provided below for de-noising the covariance works as follows:\n",
    "- The given covariance matrix is transformed to the correlation matrix.\n",
    "\n",
    "- The eigenvalues and eigenvectors of the correlation matrix are calculated.\n",
    "\n",
    "- Using the Kernel Density Estimate algorithm a kernel of the eigenvalues is estimated.\n",
    "\n",
    "- The Marcenko-Pastur pdf is fitted to the KDE estimate using the variance as the parameter for the optimization.\n",
    "\n",
    "- From the obtained Marcenko-Pastur distribution, the maximum theoretical eigenvalue is calculated using the formula from the **Instability caused by noise** part of [A Robust Estimator of the Efficient Frontier paper](https://papers.ssrn.com/sol3/abstract_id=3469961).\n",
    "\n",
    "- The eigenvalues in the set that are below the theoretical value are all set to their average value. For example, we have a set of 5 eigenvalues sorted in the descending order ( $\\lambda_1 ... \\lambda_5$ ), 3 of which are below the maximum theoretical value, then we set\n",
    "\n",
    "$$\\lambda_3^{NEW} = \\lambda_4^{NEW} = \\lambda_5^{NEW} = \\frac{\\lambda_3^{OLD} + \\lambda_4^{OLD} + \\lambda_5^{OLD}}{3}$$\n",
    "\n",
    "- Eigenvalues above the maximum theoretical value are left intact.\n",
    "\n",
    "$$\\lambda_1^{NEW} = \\lambda_1^{OLD}$$\n",
    "\n",
    "$$\\lambda_2^{NEW} = \\lambda_2^{OLD}$$\n",
    "\n",
    "- The new set of eigenvalues with the set of eigenvectors is used to obtain the new de-noised correlation matrix. $\\tilde{C}$ is the de-noised correlation matrix, $W$ is the eigenvectors matrix, and $\\Lambda$ is the diagonal matrix with new eigenvalues.\n",
    "\n",
    "$$\\tilde{C} = W \\Lambda W$$\n",
    "\n",
    "- To rescale $\\tilde{C}$ so that the main diagonal consists of 1s the following transformation is made. This is how the final $C_{denoised}$ is obtained.\n",
    "\n",
    "$$C_{denoised} = \\tilde{C} [(diag[\\tilde{C}])^\\frac{1}{2}(diag[\\tilde{C}])^{\\frac{1}{2}'}]^{-1}$$\n",
    "\n",
    "- The new correlation matrix is then transformed back to the new de-noised covariance matrix.\n",
    "\n",
    "The process of de-noising the covariance matrix is described in a paper by _Potter M._, _J.P. Bouchaud_, _L. Laloux_ __“Financial applications of random matrix theory: Old laces and new pieces.”__  [available here](https://arxiv.org/abs/physics/0507111)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note: Lopez de Prado suggests that this de-noising algorithm is preferable as it removes the noise while preserving the signal.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Constant Residual Eigenvalue De-noised Сovariance matrix is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000466</td>\n",
       "      <td>0.000288</td>\n",
       "      <td>-0.000028</td>\n",
       "      <td>0.000224</td>\n",
       "      <td>0.000252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000288</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.000025</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>0.000226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000028</td>\n",
       "      <td>-0.000025</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>-0.000019</td>\n",
       "      <td>-0.000022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000224</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>-0.000019</td>\n",
       "      <td>0.000232</td>\n",
       "      <td>0.000175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000252</td>\n",
       "      <td>0.000226</td>\n",
       "      <td>-0.000022</td>\n",
       "      <td>0.000175</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000466  0.000288 -0.000028  0.000224  0.000252\n",
       "EWG  0.000288  0.000372 -0.000025  0.000200  0.000226\n",
       "TIP -0.000028 -0.000025  0.000019 -0.000019 -0.000022\n",
       "EWJ  0.000224  0.000200 -0.000019  0.000232  0.000175\n",
       "EFA  0.000252  0.000226 -0.000022  0.000175  0.000278"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setting the required parameters for de-noising\n",
    "\n",
    "# Relation of number of observations T to the number of variables N (T/N)\n",
    "tn_relation = stock_prices.shape[0] / stock_prices.shape[1]\n",
    "\n",
    "# The bandwidth of the KDE kernel\n",
    "kde_bwidth = 0.01\n",
    "\n",
    "# Finding the De-noised Сovariance matrix using the Constant Residual Eigenvalue Method\n",
    "cov_matrix_denoised = risk_estimators.denoise_covariance(cov_matrix, tn_relation, denoise_method='const_resid_eigen', kde_bwidth=kde_bwidth)\n",
    "\n",
    "# Transforming De-noised Covariance from np.array to pd.DataFrame\n",
    "cov_matrix_denoised = pd.DataFrame(cov_matrix_denoised, index=cov_matrix.index, columns=cov_matrix.columns)\n",
    "\n",
    "# Outputting the result\n",
    "print('The Constant Residual Eigenvalue De-noised Сovariance matrix is:')\n",
    "cov_matrix_denoised"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Simple Covariance is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000466</td>\n",
       "      <td>0.000350</td>\n",
       "      <td>-0.000017</td>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000350</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>0.000303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000017</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>-0.000012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>0.000232</td>\n",
       "      <td>0.000218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000324</td>\n",
       "      <td>0.000303</td>\n",
       "      <td>-0.000012</td>\n",
       "      <td>0.000218</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000466  0.000350 -0.000017  0.000255  0.000324\n",
       "EWG  0.000350  0.000372 -0.000015  0.000221  0.000303\n",
       "TIP -0.000017 -0.000015  0.000019 -0.000009 -0.000012\n",
       "EWJ  0.000255  0.000221 -0.000009  0.000232  0.000218\n",
       "EFA  0.000324  0.000303 -0.000012  0.000218  0.000278"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('The Simple Covariance is:')\n",
    "cov_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the main diagonal hasn't changed, but the other covariances are different. This means that the algorithm has changed the eigenvalues of the correlation matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Targeted Shrinkage De-noising Method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MlFinLab also has the Targeted Shrinkage de-noising method available to users. The main idea behind the Targeted Shrinkage de-noising method is to\n",
    "shrink the eigenvectors/eigenvalues that are noise-related. This is done by shrinking the correlation matrix calculated from noise-related eigenvectors/eigenvalues and then adding the correlation matrix composed from signal-related eigenvectors/eigenvalues.\n",
    "\n",
    "The de-noising function works as follows:\n",
    "\n",
    "- The given covariance matrix is transformed to the correlation matrix.\n",
    "\n",
    "- The eigenvalues and eigenvectors of the correlation matrix are calculated and sorted in the descending order.\n",
    "\n",
    "- Using the Kernel Density Estimate algorithm a kernel of the eigenvalues is estimated.\n",
    "\n",
    "- The Marcenko-Pastur pdf is fitted to the KDE estimate using the variance as the parameter for the optimization.\n",
    "\n",
    "- From the obtained Marcenko-Pastur distribution, the maximum theoretical eigenvalue is calculated using the formula\n",
    "  from the **Instability caused by noise** part of [A Robust Estimator of the Efficient Frontier paper](https://papers.ssrn.com/sol3/abstract_id=3469961).\n",
    "\n",
    "- The correlation matrix composed from eigenvectors and eigenvalues related to noise (eigenvalues below the maximum\n",
    "  theoretical eigenvalue) is shrunk using the $\\alpha$ variable.\n",
    "\n",
    "$$C_n = \\alpha W_n \\Lambda_n W_n' + (1 - \\alpha) diag[W_n \\Lambda_n W_n']$$\n",
    "\n",
    "- The shrinked noise correlation matrix is summed to the information correlation matrix.\n",
    "\n",
    "$$C_i = W_i \\Lambda_i W_i'$$\n",
    "\n",
    "$$C_{denoised} = C_n + C_i$$\n",
    "\n",
    "- The new correlation matrix is then transformed back to the new de-noised covariance matrix.\n",
    "\n",
    "We can use this method by setting the denoise_method parameter to 'target_shrink'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Targeted Shrinkage De-noised Сovariance matrix is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000466</td>\n",
       "      <td>0.000288</td>\n",
       "      <td>-0.000028</td>\n",
       "      <td>0.000224</td>\n",
       "      <td>0.000252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000288</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.000025</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>0.000226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000028</td>\n",
       "      <td>-0.000025</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>-0.000019</td>\n",
       "      <td>-0.000022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000224</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>-0.000019</td>\n",
       "      <td>0.000232</td>\n",
       "      <td>0.000175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000252</td>\n",
       "      <td>0.000226</td>\n",
       "      <td>-0.000022</td>\n",
       "      <td>0.000175</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000466  0.000288 -0.000028  0.000224  0.000252\n",
       "EWG  0.000288  0.000372 -0.000025  0.000200  0.000226\n",
       "TIP -0.000028 -0.000025  0.000019 -0.000019 -0.000022\n",
       "EWJ  0.000224  0.000200 -0.000019  0.000232  0.000175\n",
       "EFA  0.000252  0.000226 -0.000022  0.000175  0.000278"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Finding the De-noised Сovariance matrix using the Targeted Shrinkage Method assuming alpha is 0.5\n",
    "cov_matrix_target_denoised = risk_estimators.denoise_covariance(cov_matrix, tn_relation, denoise_method='target_shrink', kde_bwidth=kde_bwidth, alpha=0.5)\n",
    "\n",
    "# Transforming De-noised Covariance from np.array to pd.DataFrame\n",
    "cov_matrix_target_denoised = pd.DataFrame(cov_matrix_denoised, index=cov_matrix.index, columns=cov_matrix.columns)\n",
    "\n",
    "# Outputting the result\n",
    "print('The Targeted Shrinkage De-noised Сovariance matrix is:')\n",
    "cov_matrix_target_denoised"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results of this de-noising method are the same as for the previous method for this particular example, however, they may differ when used on other datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### De-toning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "De-noised correlation matrix from the previous methods can also be de-toned by excluding a number of first\n",
    "eigenvectors representing the market component.\n",
    "\n",
    "According to Lopez de Prado:\n",
    "\n",
    "\"Financial correlation matrices usually incorporate a market component. The market component is characterized by the\n",
    "first eigenvector, with loadings $W_{n,1} \\approx N^{-\\frac{1}{2}}, n = 1, ..., N.$\n",
    "Accordingly, a market component affects every item of the covariance matrix. In the context of clustering\n",
    "applications, it is useful to remove the market component, if it exists (a hypothesis that can be\n",
    "tested statistically).\"\n",
    "\n",
    "\"By removing the market component, we allow a greater portion of the correlation to be explained\n",
    "by components that affect specific subsets of the securities. It is similar to removing a loud tone\n",
    "that prevents us from hearing other sounds\"\n",
    "\n",
    "\"The detoned correlation matrix is singular, as a result of eliminating (at least) one eigenvector.\n",
    "This is not a problem for clustering applications, as most approaches do not require the invertibility\n",
    "of the correlation matrix. Still, **a detoned correlation matrix** $C_{detoned}$ **cannot be used directly for**\n",
    "**mean-variance portfolio optimization**.\"\n",
    "\n",
    "The de-toning function works as follows:\n",
    "\n",
    "- De-toning is applied on the de-noised correlation matrix.\n",
    "\n",
    "- The correlation matrix representing the market component is calculated from market component eigenvectors and eigenvalues\n",
    "  and then subtracted from the de-noised correlation matrix. This way the de-toned correlation matrix is obtained.\n",
    "  \n",
    "$$\\hat{C} = C_{denoised} - W_m \\Lambda_m W_m'$$\n",
    "\n",
    "- De-toned correlation matrix $\\hat{C}$ is then rescaled so that the main diagonal consists of 1s\n",
    "\n",
    "$$C_{detoned} = \\hat{C} [(diag[\\hat{C}])^\\frac{1}{2}(diag[\\hat{C}])^{\\frac{1}{2}'}]^{-1}$$\n",
    "\n",
    "We can use de-toning setting the detone parameter to True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The De-toned Сovariance matrix is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000466</td>\n",
       "      <td>0.000288</td>\n",
       "      <td>-0.000028</td>\n",
       "      <td>0.000224</td>\n",
       "      <td>0.000252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000288</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.000025</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>0.000226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000028</td>\n",
       "      <td>-0.000025</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>-0.000019</td>\n",
       "      <td>-0.000022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000224</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>-0.000019</td>\n",
       "      <td>0.000232</td>\n",
       "      <td>0.000175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000252</td>\n",
       "      <td>0.000226</td>\n",
       "      <td>-0.000022</td>\n",
       "      <td>0.000175</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000466  0.000288 -0.000028  0.000224  0.000252\n",
       "EWG  0.000288  0.000372 -0.000025  0.000200  0.000226\n",
       "TIP -0.000028 -0.000025  0.000019 -0.000019 -0.000022\n",
       "EWJ  0.000224  0.000200 -0.000019  0.000232  0.000175\n",
       "EFA  0.000252  0.000226 -0.000022  0.000175  0.000278"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Finding the De-toned Сovariance matrix assuming the market component is 1\n",
    "cov_matrix_detoned = risk_estimators.denoise_covariance(cov_matrix, tn_relation, denoise_method='const_resid_eigen',\n",
    "                                                        detone=True, market_component=1,kde_bwidth=kde_bwidth)\n",
    "\n",
    "# Transforming De-noised Covariance from np.array to pd.DataFrame\n",
    "cov_matrix_detoned = pd.DataFrame(cov_matrix_denoised, index=cov_matrix.index, columns=cov_matrix.columns)\n",
    "\n",
    "# Outputting the result\n",
    "print('The De-toned Сovariance matrix is:')\n",
    "cov_matrix_detoned"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results of de-toning are significantly different from the de-noising results. This indicates that the deleted market component had an effect on the covariance between elements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transforming covariance matrix to correlation matrix and back"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are simple functions to:\n",
    "- transform covariance matrix into correlation matrix \n",
    "- transform correlation matrix into covariance matrix "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The correlation matrix is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.840079</td>\n",
       "      <td>-0.175654</td>\n",
       "      <td>0.775864</td>\n",
       "      <td>0.901190</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.840079</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.176822</td>\n",
       "      <td>0.752060</td>\n",
       "      <td>0.943192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.175654</td>\n",
       "      <td>-0.176822</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.132683</td>\n",
       "      <td>-0.168585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.775864</td>\n",
       "      <td>0.752060</td>\n",
       "      <td>-0.132683</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.859232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.901190</td>\n",
       "      <td>0.943192</td>\n",
       "      <td>-0.168585</td>\n",
       "      <td>0.859232</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  1.000000  0.840079 -0.175654  0.775864  0.901190\n",
       "EWG  0.840079  1.000000 -0.176822  0.752060  0.943192\n",
       "TIP -0.175654 -0.176822  1.000000 -0.132683 -0.168585\n",
       "EWJ  0.775864  0.752060 -0.132683  1.000000  0.859232\n",
       "EFA  0.901190  0.943192 -0.168585  0.859232  1.000000"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Transforming our covariance matrix to a correlation matrix\n",
    "corr_matrix = risk_estimators.cov_to_corr(cov_matrix)\n",
    "\n",
    "# Outputting the result\n",
    "print('The correlation matrix is:')\n",
    "corr_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The covariance matrix calculated back is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000466</td>\n",
       "      <td>0.000350</td>\n",
       "      <td>-0.000017</td>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000350</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>0.000303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000017</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>-0.000012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>0.000232</td>\n",
       "      <td>0.000218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000324</td>\n",
       "      <td>0.000303</td>\n",
       "      <td>-0.000012</td>\n",
       "      <td>0.000218</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000466  0.000350 -0.000017  0.000255  0.000324\n",
       "EWG  0.000350  0.000372 -0.000015  0.000221  0.000303\n",
       "TIP -0.000017 -0.000015  0.000019 -0.000009 -0.000012\n",
       "EWJ  0.000255  0.000221 -0.000009  0.000232  0.000218\n",
       "EFA  0.000324  0.000303 -0.000012  0.000218  0.000278"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The standard deviation to use when calculating the covaraince matrix back\n",
    "std = np.diag(cov_matrix) ** (1/2)\n",
    "\n",
    "# And back to the covariance matrix\n",
    "cov_matrix_again = risk_estimators.corr_to_cov(corr_matrix, std)\n",
    "\n",
    "# Outputting the result\n",
    "print('The covariance matrix calculated back is:')\n",
    "cov_matrix_again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Exactly the same as the original one:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EEM</th>\n",
       "      <th>EWG</th>\n",
       "      <th>TIP</th>\n",
       "      <th>EWJ</th>\n",
       "      <th>EFA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>EEM</th>\n",
       "      <td>0.000466</td>\n",
       "      <td>0.000350</td>\n",
       "      <td>-0.000017</td>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWG</th>\n",
       "      <td>0.000350</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>0.000303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TIP</th>\n",
       "      <td>-0.000017</td>\n",
       "      <td>-0.000015</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>-0.000012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EWJ</th>\n",
       "      <td>0.000255</td>\n",
       "      <td>0.000221</td>\n",
       "      <td>-0.000009</td>\n",
       "      <td>0.000232</td>\n",
       "      <td>0.000218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EFA</th>\n",
       "      <td>0.000324</td>\n",
       "      <td>0.000303</td>\n",
       "      <td>-0.000012</td>\n",
       "      <td>0.000218</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          EEM       EWG       TIP       EWJ       EFA\n",
       "EEM  0.000466  0.000350 -0.000017  0.000255  0.000324\n",
       "EWG  0.000350  0.000372 -0.000015  0.000221  0.000303\n",
       "TIP -0.000017 -0.000015  0.000019 -0.000009 -0.000012\n",
       "EWJ  0.000255  0.000221 -0.000009  0.000232  0.000218\n",
       "EFA  0.000324  0.000303 -0.000012  0.000218  0.000278"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Exactly the same as the original one:')\n",
    "cov_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook describes the functions implemented in the RiskEstimators class, related to different ways of calculating and adjusting the Covariance matrix. Also, it shows how the corresponding functions from the mlfinlab library can be used and how the outputs can be analyzed.\n",
    "\n",
    "Key takeaways from the notebook:\n",
    "- A robust covariance estimator (such as the Minimum Covariance Determinant) is needed in order to discard/downweight the outliers in the data. These outliers seriously affect the Empirical covariance estimator and the Covariance estimators with shrinkage.\n",
    "- The Maximum Likelihood Estimator (Empirical Covariance) of a sample is an unbiased estimator of the corresponding population’s covariance matrix.\n",
    "- Shrinkage consists in reducing the ratio between the smallest and the largest eigenvalues of the empirical covariance matrix. It is used to avoid the problem with inversion of the covariance matrix.\n",
    "- Ledoit-Wolf and Oracle Approximating are methods to calculate the optimal shrinkage coefficient $\\alpha$ used in the Basic Shrinkage.\n",
    "- The semi-covariance matrix is the way to measure the volatility of the negative returns or returns below a certain threshold. \n",
    "- Exponential Covariance is used to give greater weight to the most relevant observations in computing the covariance.\n",
    "- The Constant Residual Eigenvalue De-noising Method calculates the eigenvalues of the correlation matrix and adjusts the ones that are lower than the theoretically estimated ones, as they are caused by noise.\n",
    "- The Targeted Shrinkage De-noising Method shrinks the eigenvectors/eigenvalues that are noise-related.\n",
    "- The De-toning algorithm excludes a number of first eigenvectors representing the market component. This allows a greater portion of the correlation to be explained by components that affect specific subsets of the securities. Note, that a de-toned correlation matrix cannot be used directly for mean-variance portfolio optimization."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mlfinlab",
   "language": "python",
   "name": "mlfinlab"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
